{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Overview\n",
    "- **Expectation Used:** [expect_column_values_to_be_equal_to_or_greater_than_profile_min](https://github.com/great-expectations/great_expectations/blob/develop/contrib/capitalone_dataprofiler_expectations/capitalone_dataprofiler_expectations/expectations/expect_column_values_to_be_equal_to_or_greater_than_profile_min.py)\n",
    "- **Expectation Description:** This expectation will take the report of an initial dataset and compare it to the report which is generated by an additional dataset of the same schema. The expectation is that the user specified column should contain values greater than or equal to the min value metric generated in the report of the initial dataset.\n",
    "- **Example Details:** In this example, the data owner is using this expectation to compare two different time series in the dataset they are using. They want to ensure that all salaries at their company are greater than or equal to the `min` value generated in the report from the original year."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Imports"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Great expectations imports\n",
    "import great_expectations as ge\n",
    "from capitalone_dataprofiler_expectations.expectations. \\\n",
    "    expect_column_values_to_be_equal_to_or_greater_than_profile_min \\\n",
    "    import ExpectColumnValuesToBeEqualToOrGreaterThanProfileMin\n",
    "from great_expectations.self_check.util import build_pandas_validator_with_data\n",
    "\n",
    "# Data Profiler import\n",
    "import dataprofiler as dp"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Setup\n",
    "Below we are going to import a dataset from the Data Profile testing suite. This csv holds information on the salaries of individuals in the data science field from all over the world."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "context = ge.get_context()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "data_path = \"../../dataprofiler/tests/data/csv/ds_salaries.csv\"\n",
    "data = dp.Data(data_path).data\n",
    "data"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In this example we are going to split up the dataset into three separate years so we can simulate a dataset which will have a yearly aggregation of data."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "data.sort_values(by=\"work_year\", axis=0, inplace=True)\n",
    "years = data[\"work_year\"].unique().tolist()\n",
    "years"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now that we have the years, we will capture all records from each year in their own dataframes, so we can process them separately."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "individual_dataframes = []\n",
    "for year in years:\n",
    "    current_year_df = data=data.loc[data[\"work_year\"]==year]\n",
    "    current_year_df = current_year_df.drop(\"work_year\", axis=1)\n",
    "    individual_dataframes.append(current_year_df)\n",
    "individual_dataframes[0].head()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we will create a report on the first `individual_dataframe` which corresponds to the year **2020**, then we will output the `min` metric from the `salary_in_usd` column as found in the report."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "profiler_options = dp.ProfilerOptions()\n",
    "profiler_options.set({\"data_labeler.is_enabled\": False})\n",
    "profile = dp.Profiler(individual_dataframes[0], len(individual_dataframes[0]), options=profiler_options)\n",
    "report  = profile.report(report_options={\"output_format\": \"compact\"})"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's take a look at the output `min` metric from the `salary_in_usd` column as found in the report"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "report['data_stats'][6]['statistics']['min']"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Running the Expectation\n",
    "We build the validator by passing in the `individual_dataframe` corresponding to **2022**. Then we will use the exception below to find that there are no values in the `salary_in_usd` column that exceed the `max` metric for the `age` column generated in `report`."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "validator = build_pandas_validator_with_data(individual_dataframes[1])\n",
    "results = validator.expect_column_values_to_be_equal_to_or_greater_than_profile_min(\n",
    "    column='salary_in_usd',\n",
    "    profile=report\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Results\n",
    "After the data owner generates the expectation results they find that there are 5 salaries recorded in **2022** which are less than the `min` `salaries_in_usd` recorded in **2020**. Despite that the expectation failed and there are some salaries which are lower two year after the original report, the expectation results still indicate which salaries are lower the data owner can tell the company where it might need to compensate more."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "results"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
